1. Technical Field
The invention relates to modeling and simulating software systematically. More particularly, the invention relates to a system and method for modeling, simulating and augmenting software, its user intelligence and their interaction based on software dynamic system representation techniques.
2. Description of Prior Art
The fundamental principle of software is to model and simulate the processes of human thinking, doing and living, including some that have not existed before. In the quest to understand the world around us, every professional rank turns increasingly to the software for modeling and simulation. Whether phenomena are natural or artificial, scientific or entertaining, physical or mental, they all can be modeled and simulated by software. Software is already being used to model and simulate the tiny forces binding molecules, the flights of aircrafts, the creative imagination of artists, the stability of machinery designs, and the behavior of complex systems. Instead of engaging in the actual, physical process to study the underlying phenomena, models are identified and created to substitute the real ones and conducted as a simulated process on computers without the constraint of cost and feasibility required to run a physical process. Software becomes a universally accepted representation of all kinds of domain knowledge. Software developers strive to develop better models for their professional fields and to improve efficiency of software operations, yet still must struggle to find a representation mechanism for the software itself.
The majority of software built and released belongs to the category of interactive software that requires user's intelligence to operate and control. Despite the advancement in user interface technologies, running the software interactively still poses a big challenge that demands the initiative, the selection, the interpretation, and the decision for the user to overcome. For example, a task running Microsoft® Excel to optimally allocate business resources involves the intimate knowledge of linear programming principles as well as the skill of molding that knowledge into the crowded Excel user interface (FIG. 1A). For a novice user 10, he may stumble at any stage 12 for the lack of understanding either the domain or the software operational knowledge. On the other hand, an expert user 20 commanding the same software and the task may attain his goals 22 skillfully (FIG. 1B).
In stark contrast to the software's origin from a rigid modeling and simulation of universal phenomena, software and its user intelligence and their interactions hardly are investigated systematically in a commensurable way. Instead, the software industry resorts to traditional means, such as books, manuals, videos, or multimedia slides and presentations, to represent the most complex and challenging intellectual process that happens right on its own turf.
The method widely used today to simulate a software operation is to capture or record and replay screenshots for a multimedia slide show.
To capture the screenshots, it requires the software to be controlled by a human user or other means to perform some tasks.
One that is based on typical manual capture and edit methods is well documented in a U.S. patent application Ser. No. 10/121,117. It requires a human to manually operate the software while the software display on screen is captured manually. After the screenshots are captured, it goes through a manual authoring process to put together a multimedia slide. The primitive nature decides that it can only author some basic graphic user interface operations such as selecting a menu, clicking a button, and so on in a short multimedia slide show, which may last only tens or hundreds of screenshots.
As software is running on computers, it is natural to develop an external program or script to automate the tasks. The next one that is based on the technique used extensively in software automatic test field and adapted later for capturing screenshots can be best represented by Microsoft® Test product and varieties in the art earliest since the 90 s. First, it controls the software being captured by applying some synthesized input commands through a script to drive the software into a specific state. Next, the software display is captured by taking a screen image. Finally, the captured screen image is compared with a baseline screen image that is known to be valid. It runs through the process to capture and compare multiple frames of screen images. The method adapted for capturing screenshots uses the basic steps above except forgoing the final step of comparison.
No matter what methods, manual or scripted are used to capture screenshots, the final product as produced in the art is a video for replaying. However, by treating the software as a visual effect without a mechanism only capable of churning out those colorful bits, the essential system information that is powered by the software, its controlling host and their interaction is lost permanently. Without any model or structure that enables the interaction between the software and its controlling host being modeled, identified and simulated, so much of the programmatic events and actions embodied in the underlying process all degenerate into unrelated dumb bits.
The techniques to control the software by an external program as a temporary means to control the software in order to capture screenshots are not new in the art. The external controlling program used during capture is discarded and no longer serves any further purpose in replay. Therefore, the control structure that facilitates the interactive dynamics between the software and the controlling program is destroyed irreparably.
Still, the ad-hoc nature of these screenshot methods prevents the methods from capturing and simulating highly complex interactive processes between the software and experts. For example, using a CAD software to design a machine, an animation software to master a 3D title, it involves not only a complex software but also complex strategies from experts to drive the software to attain goals. The strategies often involve tedious but precise human motor movements in the software work-space. The process may take tens of thousands or even more screenshots to record in a temporal dimension. It may also demand accuracy to perform functions within a range of a few pixels on screen in a spatial dimension.
The methods performed by current art concentrate on showing simple user interface features, procedures and functions of the software, but ignore the fact that the software and its controlling host is a holistic unity, and the use of the software to perform a task in an interactive process is just as important as the use of tools to control phenomena which is modeled and simulated by the software already.
Without a structure and a mechanism to represent the interactive process holistically, current art is unable to model and simulate the complex temporal and spatial relationships embodied in the process. The ways of designing a machine, a chip, a 3D title or using software to perform a job, are still mainly locked in the minds of those highly intelligent experts.
Thus, it is clear that the fundamental missing in current art is a framework that can systematically represent the software that runs on digital computers, its user who controls the software, and a dynamic process that interacts between them. If the software is in the business of modeling and simulating every conceivable natural and artificial phenomenon, then why can it not heed its own calling to model and simulate its very own cause—the software, its best user intelligence, and their interactions in software. Therefore a software representation system that can model and simulate the software systematically just as the software can model and simulate its underlying domain, is highly desirable.
There is a research field in system science called Hybrid Dynamic Systems that is studied extensively in academia. The central goal for research is to bridge the gap between computer science and systems science by developing the foundations of a modern systems science that is simultaneously computational and physical. One of the studied topics is modeling the software as a component in a hybrid system, in which physical processes are controlled by control algorithms that are implemented in the software. Unfortunately, the hybrid systems only address the interaction between the software and conventional physical processes, not between the software and its human users.
Also, without the system framework to represent the software, a huge intellectual value that is built into the software is unutilized. More than trillions of dollars worth of software that has touched all possible humankind endeavors has been introduced over the last quarter of the century. Software contains content embodying the underlying domain knowledge that is programmed and distributed in the binary formats of EXE and DLL files. Unlike traditional artificial intelligence in which people refer to knowledge in the forms of documents, e-mails, databases, and conversations with a human expert that have static textual expressions, the knowledge embodied in software is binary as well as tacit, intangible, dynamic, and of an interactive computational process, which is impossible to be represented by current art.